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ABSTRACT 

Project objective for the second year of the Computer 
Based Project for thei Evaluation of Media for the Handicapped was 
development of a functional system for evaluation. The evaluation 
system was intended to be neutral to both type of material being 
avaluated and population of handicapped chiidren. Development of the 
functional system for eTOluation of media was supported by 
establishment of a data bank of media and student characteristics, 
inservice teacher training for six persons on principles of 
Instructional technology, discussions and work sessions held with the 
staff of the Evaluation Unit of the Buffalo special Education 
Instructional Materials Centers, and information dissemination 
activities. Administrative procedures for collecting, analyzing, and 
storing data have been developed and modified according to needs. 
Research findings and questions generated included; effects of 
pretest and posttest design; automated versus nonautomated testing 
situations; effects of captioning and audio stimuli; and other 
related Investigations, Appended were dlagramatic models of the 
evaluation system- (CB) 
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PROJECT OBJECTIVES 

The Computer Based Project for the Evaluation of Media for the 
Handicapped has completed its second year of activities. 

The project is conceived as meeting a portion of the committment 
specified in Bureau of Education for the Handicapped Objective number 3, 
i.e., ''By 1976, provide systems and vesources so that significant and 
relevant materials are readily available to all teachers of handicapped 
children so that at least 601 of the handicapped children will be served." 

One contribution of the Computer Based Project is to be that of 
providing a system of materials evaluation to produce information on the 
effectiveness of the materials In Media Service and Captioned Film 
Depositorios with given populations of handicapped children. This evalua- 
tive Information may be transmitted through the IMC/RMC t'letv.'ork to give 
educators Information with which to make maKimum utilization of the 
materials. The evaluation system Is to be essentially neutral tot 

1. the type of material being evaluated 

2. the population of handlcappod children 

The development, tryout, and validation of the evaluation system 
and the resultant data banks of student characteristics and media 
characteristics are seen as a vital part of a long term, larger encompass- 
ing objective of improvement and Ind ividuel Izat ion of the ©ducatlon of 
handicapped children initially In the Syracuse City School System with 
eventual dissemination of information nationwide. The evaluative Inf or- 
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teacherSp as input for moro appropr ‘ate Instructional rloclsionsj 

2. materials producers^ as input for design of materials; 

3. admi n I strotors 5 as Input for management decisions tind procGdures 
for providing Instruction for handicapped children; 

4 . researchers^ as data for curriculum devolopmant, Instructional 
design, learner characteristics, and rther relationships. 

MAJOR OBJECTIVE 

The major objective of the second year of the project was to develop 
a functional system for evaluating media for the hand icapped , All rosourc 
during the second year were channeled Into accomplishing this task. The 
successful achievement rf this major objective required a mcl t i "cemponent 
pirogram -- each of which was otscntlal to the successful development of 
the evaluation system. 

/ The tentative evaluation model developed by the project during the 
first year underwent Intensive examination, testing, modification and 
clarification during the second year. This Involved: 

1. application of the tentative evaluation model In the review of 
existing media and the preparation of the media for the 
evaluation process; 

2. expanding the Student Response System to twenty stations at 
the Center in order to facilitate Its use by whole classrooms 
of children; and 

3 . utilizing research findings for decision making related to the 
evaluative process I I.e,, automated Instructional system vs 
traditional teaching, use of pre/post test design, caption vs 
non-caption, and audio vs non-audio. 

The major objective was supported by the following components: 



ijata bank component 



The evaluation system was supported by the establ tshment of a 
data bank of media and student characteristics, This information served 
as an essential component In the decision making process related to the 
media as well as supporting the Information collected from the Student 
Response System. Activities related to this component Involved: 

1 , collecting, storing, and analyznng data related to characteristics 
of students and media, and student responses to media; 

2, outlining procedures for using the d;:ta bank; 

3, analyzing computer capabilities needed for storing and 
processing the data and supporting the evaluation model; 

A. determining, via the data bank, the cost effectiveness of 
Student Response System and the evaluation model; and 

5. using the data bank to make grouping and materials decisions, 
IN-SERVICE COMPONENT 

As part of the Computer Based F^roject Evaluation Model, It was 
deemed essential to train teacheirs to participate In the field testing 
(Trial C) of the processed materlalls throughout the handicapped program 
in the city. Special Education Traachers were trained: 

1 . to preview and prepare objectives and questions for media 
boing processed by the evaluation model; 

2. to utllizo techniques of programmed Instruction by: 

a. proper use of Instructional media In the present curriculum, 

b. reporting evaluative data on the use of materials, 

c. defining behavioral objectives, and 

d. diagnosing student progress, 

f 
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I NTER-^ROJECT COMPONENT 

The Computer Based Resource Units and Evaluation Project at 
Buffalo and the Computer Based Project In Syracuse have both been involved 
in the evaluation of materials anc! the uso of the computer for storaQS, 
analysis, and retrieval. Therefore, an essential ingredient In develop- 
ing the functional evaluation system Is to formulate a model for Incorpora- 
ting the extensive Information accumulated In Buffalo and the intensive 
Information developed in Syracuse Into a system which will provide feedback 
to both models. 

DISSEHlUATiOM C OMPONENT 

Since evaluation of Instructional media Is the concern of all project 
directors, it was desirable for the Computer Based Project to receive feed- 
back from other project directors as to the relevancOi appropr lateness , 
off ect I veness , ease of utilization, etc, of the Computer Based Project's 
tentative evaluation model. In acidltioFi, the Intense focus of the project 
on materials ovaluatlon provides not only formats for evaluating existing 
materials, but lends Information relative to the freparatlon of now materials 
The dissemination component, as such was and Is ossentfal in assisting the 
project in perfecting a useful and functional evaluation system. This 
Initial step was accomplished by: 

I* sending quarterly and annual reports to all Hodia Services and 
Captioned Films Project Oi rectors | 

2. preparing evaluative Information on several MS/CF films and 
filmstrips for dissemination to teachers In the Syracuse City 
School s I 
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assembling a package of supnoptlvo materials for each media 
evaluated consisting of objectives and criterion quostlons 
for Its use; 

4, preparing a handbook on procedures and techniques for using 
Computer Based Project Model for evaluating media; and 

5. preparing a list of recommendations relativo to production 
of new materials, 

QBJCTSVE I 

" to develop a functional system for evaluating media for 

the hand Icapped 

The first year had produced a tentative evaluation system and 
some experiences Involving children with media. Several shortcomings 
wore identified and corrections made to somo obvious problems. 

The functional system developed during this second year of operation 
is still capable of further refinement for cast-effectiveness and the 
Inclusion of various forms of Instructional media other than film and 
f I Imstrips. 

The Third Quarterly Report^ dated March 31* 1971 » contained a 
description of the tentative evaluation model. This model is 
continually being evaluated and modified, 

A flow chart of Its present stats Is In Appendix A. 

Areas that have been defined In the third quarterly report and 
remain without change will not be repeated In this report. 

The first change In the model has been In the selection of media for 
evaluation and In the personnel assigned to that task. 

Materials on file are placed In the evaluation process on a priority 
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basis. The priority system was described in our second quarterly report 
dated December 31, 1970 « However, even it has had to be modified. The 
priority system now allows a staff member to preview a film and fill in 
the checklist of information on the material being examined. One piece 
of information is his professional judgement for priority of evaluation. 
This judgement is recorded on a scale from "A", high to ”D" low priority. 
In making this judgament ha considers content appropriateness, method of 
presentation, and overall effect. If a film is rated "A", it proceeds in 
the eveiluative process. If it does not recei%-e an "A", it goes to a hold 
file where, when the available '’A" films are depleted, it is subject to 
reclassification. The other information collected in this critical view- 
ing is also recorded on the priority classification checklist and is 
punched on data cards to be used for scheduling and other administrative 
tasks . 

A piece of instructional material that has received a priority 
elasslf ication of A goes on Into Trial Attention Observation (TRIAL AO) , 
which is also a new step in the evaluation process. Trial AO is a small 
sample, field based intaraction of student, media, teacher, and tester. 

It has replaced our old Trial A experience out in the schools and has a 
different purpose. The purpose of Trial AO is to see if the students are 
Involved enough with the neditmi to permit us to expend further time and 
energy on it. If they are, their responses give insights as to their per- 
ceptions of the key parts of that material. 

The field testing techniques developed at the project and reported 
in the first quarterly report are essentially the same with the exception 
of the lack of pre-testing and post-testing of students. 



Experience will dictate exact sample size for Trial AO^ but at 
this time plana are for three classroom unit observations at different 
age and handicap levels. 

Five data bases will be collected s 

1. Student attention data 

2« Student Interview data 

3, Student responses to open-ended questions on a group setting 

4, Teacher interview data 

5, Field tester data. 

Data base three is the only new data base and will be used primarily 
for Iriaights for the question writer in preparation for Trial A. 

At this point, a decision is made on whether or not to continue a piece 
of material in the evaluation process. If the decision is YES, the material 
is prepared for Trial A. If the decision is NO, a final report is t^'ltten 
on the material using ns much Information as is already available. The 
criterion for the continuation of an instructional material In the process 
is presently being evaluated, but tentatively a 67,5% Binimum level of 
student attention is being used, (75% level of the students watching 90% 
of the time.) 

The process of preparation for Trial A has not significantly changed 
but significant administrative changes have been made which should be 
mentioned. Whereas previously there had been three instructional specialists 
sharing this and other rasponsibilltles , there Is now one person responsible 
for the development of all evaluation Instruments, This change has been 
made for two basic reasons. First, by devoting more of one’s time to one 
basic area, It would seem reasonable to expect more depth and expertise in 
the approach and consequently a better product. Second, by assigning one 



person this responsibility ^ accountability for Instrument development 
and quality is more clear. 

The process of preparation of questions for a piace of inaterlal night 
be best eKplnincd by the use of an CKample, 

This process Involvee determining the message of a films isolating 
the facts, principles , and ideas by time segments, determining which facts 
are appropriate at various levels of student performance , writing behavioral 
objectives for specific levels of students, determining the type of question 
style to be used, and writing questions to specific cells of the STRUCTURE 
OF THE INTELLICT, The question written may be related to spec7.flc time 
segments of the medium. 

The questions developed can he asaembled into a test instrument which 
can probe segments of the sample population such as by age or by type of 



Following is an ex^ple of this technique applied to the film 
"Aninals In Winter,’ 



GEHERALIZATIOili Animals live through winter in different ways — — - 



PRll’IARY 



INTERMIDIATE 



SECOl'JDARY 



FACTS s FACTS j FACTS ; 

1. Some animals are friends. 1, Some anitnals change 1, Feathers keep 

colprs In winter, birds warm. 

2. Some animals are eu' jiras, 2, Some animals sleep 2, Moths live 

moat of the winter. through winter 

as a pupa. 



3. Some anlnalo live in dens. 3. Some animals sleep 

and hunt duiing 
the winter. 



3 , Animals like 
woodchuck and 
chipmunk hiber- 
nate during 
winter. 



4. 




Somo birds don't fly south.. 4, 




A caterpillar or 
moth can not live 
through winter. 
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4, Rabbits stand 

still when danger 
is near* 



IHTBE1'®IATE 



SEOOflDARY 



PRUIAEY 



BEHAVIORM. OBJECTIVES 

1 . Gi'/en thrae visuals , 
the child will select 
one that shows atilirials 
are frleiids. 



2. Given thrae visuals 9 
the child will salect 
one that shows animals 
are enemies. 



3, Given a visual/audlo 
question the child 
will identify a bird 
that docjn't fly 
south, 

4. The child will identify 
a visual of a woodchuck 
as a dan. 



BEHAVIOML OBJECTIVES s 

The child will Ideiitify 
the followlno by respond- 
inp, correctly to multi- 
choice questions i 
1 , Animals change 
color in winter. 



2. Animals sleep 
in winter. 



3. Some aniDala sleep 
and hunt during 
winter. 



4, A caterpillar or 
moth cannot live 
through winter. 



BHIAVIORAL OBJECTIVES? 

1, The child will 
define pupa as 
being a stane 
in the life cycle 
of a moth. 



2, Ttie child \ 111 
define hiberna- 
tion as a sleep 
like state of 
some animals 
during winter, 

3, The child will 
eKplalii why a 
rabbit; stands 
still when 
danger is near. 

4, The child will 
GKplain why 
feathers keep 
a bird warm. 



Questions written to 
cells of Structure Of 
llic Intellect for Primary 
Behavioral Objectives, 



Questions written Questions written 

to cells of Structure to a cell of 

Of The Intellect for Structure Of The 

Intermediate Behavioral Intellect for 

Objectives. Secondary Behav- 

ioral Objectives, 



Memory Flgural Units 



Memory Semantic 
Relations 



Convergent 
Semantic Classes 



M®aory Symbolic Units 



Cognition Semantic 
Relations 
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The present Trial A has the same objecwlve, to eliminate teat items 
^hich do not measure behavioral change ^ as the old Trial A but the setting 
is different. Instead of being conducted in the schools it will be conducted 
at the Conputar Based Project Center with additional studants being bussed In 



to utilise the Student Response System. This change was necessitated by 
reactions of the students, teachers, and staff to the problems encountered 



by using untested questions in the schools, not reinforcing correct post test 
answeis, and general difficulty in gathering reliable field test data. 

Output will be an attention profile, results on the pre/post test, 
interview data, and anectodal comments. During the processing of the pre/ 
post test, the computer will reject those questions which fail to satisfy 
the project criterion for question acceptability, 20% gain and/or 80% prior 

knowledge by age and handicap level. 

The material is then formated for Trial B hy eliminating those rejected 



questions and making any other preparations necessary. A Trial B sample is 

same manner as for A and the same testing procedure is used, 



assignea in 



The purpose of Trial B is different, however. Trial B, using the instrument 
developed In Trial A, is to evaluate the media. In B, the media is the 



prime subject of examination, not the questions. 

The instructional specialist in charge of technical writing will receive 



a folder < 
attention 
Also aval 



data from Trial B, In this folder will be pre/post printouts, 
□file. Interview results, and anectodal comments from Trial B. 
(le will be all the other data on the material under examination 



that the Specialist must make evaluative statements about under the following 
categories I objectives, recomended usage, attention data, interview data, 
teachers* evaluation, questions, racomraended question list, descriptors. 



and vocabulary 



An example of this writeup was Included iu the Third Quarterly Report 
but modifications are anticipated during the next quarter. 

The report at this point represents a close look at student interaction 
with instructional materials and their rGaponses to it. One weakness of 
this close look, however, is the eli mi nation of two key factors in the 
educational process: the student in a "Normal" classroom setting and his 

teacher. These factors necessitate a Trial C or field test phase. The 
purpose of Trial C is to validate our media evaluation statements in that 
"Normal" classroom setting with the teacher. 

There are three key elements in this field test phase. First, the 
teacher muat find something in the report that makes her feel that it Is 
appropriate for her students. Second, she must present it to her class 
and/or student and respond to the questlon’iaire. Third, she will be 
encouraged to have the child utilize a post tost instrument selected from 
our report and/or her own instruraent. 

This data will be returned to the center for processing. After a 
sample at each age and handicap level has been accomplished, the data will 
be surveyed and compared to the media evaluation report. Consistencies 
in Trial A and B performance and Trial C results are hoped for. Disparity 
in results will require a check of Trial B. interpretation. If still thought 
correct a larger Trial C sample will be run. If disparity continues re- 
evaluation of the material may be necessary. 

The final evaluation report will be submitted to MS/CF, the Buffalo 
Computer Based Resource Units Project, and disseminated to others on request. 
DATA BMiK COtgQNENT 

At the present time, the project has on file seventeen characteristics 
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on each of 1565 children in .special education classes, has established 
files on 404 films, 157 filmstrips, and has recorded approKimately 10,000 
student responses to questions on media, 82 attention observation profiles, 
and Interview data from the students. A data analysis breakdown for the 
year is in Appendix B, 

Attention has been directed to analyzing student responses to media 
in order to develop format and information which will allow the tentative 
evaluation report to be optimal. Format for student characteristic data 
analysis and its relationship to student performance has been eBtablished 
but the actual analysis of the data has been delayed due to computer facility 



Procedures for the use of the data bank have been developed and 
are an ongoing activity as the sophisticatir a of the data bank Increases, 

A variety of computer capabilities have been Investigated and were 
the subject of a position paper appended to the recycling proposal for 
1971-72. Constant analysis of the demands and resources of the project 
will be necessary, 

IN-SERVICE COtff ONENTS i 

^ core of 6 teachers were given background training in the principles 
of Instructional technology and became Involved on an after school basis 
with the project activities. Their primary function was the development of 
items for testing the effectiveness of medial but over the course of the 
year more innovative attempts at using Instructional materials in the class 
room were noted. A selected group of 5 additional teachers were utilized 
to experiiaent with approaches to mediated Instruction preliminary to rrial C 
and further training of larger groups of special education teachers. The 
fornmt for this In-service was innovative and proved to be successful. The 
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project contracted with the teachers for a behavioral objective that they 
Identified as Important to their teaching and provided resources, support, 
supervision, and evaluation along the way for her achievement of the 
objectives. This eKporionco was not only beneficial to the teachers but 
Increased the porformance level of the students and was extremely ’•easonable 
In terms of cost, (The teachers put In three to four times more lours than 
were contracted for and with no hesitnney or griping.) The removal of the 
role of the In-service director as a dispenser of Information and establishing 
a position more as a supportar and a manager of Instruction were very useful 
and successful. 

INTER PROJECT C0HP0NEM7 

Discuss ions and work sessions were held with the staff of the Eva’ *atlon 
Unit of the Buffalo SEIMC. Further clarification and development will be 
necessary but there appear to bo no Insurmountable problems. 

The Evaluation Unit of the Buffalo SEM1C has classified all MS/CF 
filmstrips using their categorlos and the Computer Bas^ d Project is checking 
those classifications with the validated objoctivas tested at the project. 



DISS EMI MAT ION COMPONENT 



The ipccffic dissemination activities of the project Includedi 

1, Sending 3 quarterly reports to the project mailing list 
which Includes other MS/CF project directors. 

2, Preparing evaluative Information on each film and filmstrip 
that has been through the evaluative process, 

3, Assembling a package of appropriate materials for each 
madia evaluated for distribution to classrooms. 
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The data processing manual which specifies procedures 
and techniques for using the model in evaluating media 
has been prepared in draft form, 

5, Presentation of papers at national and International 
conventions. (See Appendix C) 

The project did not sponser a Media Evaluation Workshop for all 
Project Directors this year and did not prepare a list of recommendations 
relative to the production of new materials due to the availaljility of 
existing position papers which adequately express the project’s opinion. 

SUPP ORT ACTIVITIES AMD F INDINGS 

ADMINIITMPVE 

Project Staff have been involved in several activities to support the 
attainment of specif ied objectives that may be significant to the readers 
of this report. 

Administrative organization and procedures have developed along with 
the Project. 

EQUIPME NT IIArEMTORY 

An effective equipment inventory procedure has been developed which 
allows for maxlmmri usage of the equipment but also accountability for its 
location and malntonance. 

FILING 

Procedures for filing of reports on the instructional materials have 
been developed and Implemented. A catalog of films was developed and distri- 
buted to the special education classroom teachers in the city school district 
preparatory to the usage of the materials in their classrooms. 



DATA COLLECTICm 



Administrative procedures for collecting, analyzing, and storing 
the data collected have been developed and modified as needs Tnatorlalized , 
p^Q'^2,Qitis rcsultltlS lost or unuseable data, have been identified and 
rGCtifled for neKt ycar-s effort. 



One of the largest problem areas was in tho gathering of information 
from young children not fully capable of responding to verbal and/or written 
questioning techniques by indicating the desired an.! 



.Rt'jer on an answer sheet. 



Modifications of response sheet format and evaluations of skill level of 
tliGi r.hi l.druii indicated the scope of the problem. Crude training programs 
were instituted with some success but the area most subject to review is 



the quostionlng technique Itself. 

Several alternatives have been Idontlfiod and experimented with on a 
limited basis. One such alternative is tape recording a group discussion 



about the movie after its showing. Another alternativa is the construction 
Qf a, group story about the movie in the same fashion as the morning Classrootn 
News” is done in many classrooms. The students follow up this activity by 
drawing a picture of their perception of the film and then xdentlfy what 
they have drawn to the teaGlier'. Tliis technique offers some exciting potential 
to the evaluation process . 

One other form of data that has not yot been fully devoiopod but that 
has potential is the Interview format. Presently, the project is asking 
children simple questions like ”Did you like the filiri?” and ”How much did 
you learn?" Techniques such as this may permit more in depth and valid 
responses and help identify some affective dimensions of the instructional 
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C LASSROOM RELATIOHS 



The whole areas of relationships of the project with the classroom 
toacher has come under analysis aome changes have been institutod, 

TGacher foadback has indlcatod that: teachers desire regular feedback on 
rosults of progranis that their childran arc mvolvod iii and advanco not ice 
and vGto power on niatcrials selectod for showing* Bie intoresting phenO"” 
menon here is that they did have both of these powers all year long but 
gonerallyr failed to OKcrclse thorn. The biggest discussion has beon on 
curricular rGlevancy of the tnatcrlals being evaluated* It is the fesllng 
of the project that these inaterlais should be evaluated^ at least initially 
in a neutral environment J that is, evaluatod Inciopondont of 'good usage 
or "bad" usage of instructional niaterlals . 

The teachers have been given a copy of the questions asked on a 
piece of material at the time of the classroom visit. Ho formal follow-up 
has been made on the usage of these materials but general indication from 
the field testors indicate that only a few teachers take advantage of this 
material and Integrate it into their program. These do appear to have 
public relations value for most toachers, hoT weri The specific results 
of the feedback from classrooms involved is incltidGd with the questions 
In Appendix D. 

These indications have helped to change the process as indicated 
in the first portion of this report. 

JOB DESCRIPTIO NS 

As the development of the tnodol progressed , the neod for staff member 
to function as a team with specific Job responsibilities became apparent. 



VJhereas the project had operated effectively in the developmental stasos 
with shared responsibility areas, the increase in quantify of matorial and 
depth of approach taxed that organization. Next year, the praviously 
shared responsibilities of question dovelopmant , result intGrpertation, 
technical writing, sdiool relationships, testing supervision, curricular 
integration, and dissomination will be assigned to apecific staff members 
rather than shared* This does not prsompt one staff member from aslvlng 
another’s assistanca but it does pin point accountability and lesson the 
confusion of job responsibilities for each staff member. 

STUDE NT RESPO NSE EQUIPMENT 

The unique equipment prepared by General Electric's Research and 
Development Corporation for the project 's usage has been very satisfactory 
after some preliminary problems with reliability. Tliese problems should 
be expected with equipment In a development state and the cooporation that 
the project rccelvod has been outstanding In the resolution of chose problems. 
Specifically, at one tlmo in the winter the project's twenty SRS stations 
ware blowing Integratad circuits on an alarmingly regular basis with no 
apparent cause. The magnitude of this problem was of concarn both to the 
project and General Electric. After chocking several standard possibilities, 
such as voltage fluctuation, short circuits, etc. it was discovered that the 
problem resulted from the children walking across the carpeted floor with 
the relative humidity being low. The resultant static electrical shock 
when the children touched the responder was sufficient to blow out the 
intograted circuit and cause a malfunction. Grounding the stations, spraying 
the carpet with "X-statlc," and the purchase of a htmidlfler solved the 



problem. 
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An additional observation of the affect of the SRS equipment on student 
performance came at the beginning of the school year. Delivery of five new 
stations had been delayed and some students had to sit at desks without 
reaponders and answer with pper and pencil. The disappointment of these 
children became more obvious as time went by. Finally, to restore the 
positive feelings of the group without stations, non-working stations were 
placed on the desks and the children were tol .hat they were working. 

The Audio/Visual RGsponder unit developed by General Electric has 
become a very, effective tool in the control and evaluation of non-supervlsed 
presentations of programned materials and can also be utilized with non- 
programmed materials. The project uses it as a monitoring devica for film- 
strips and Project LIFE evaluations# In addition to the monitoring function 
an audio recording of the captions on filmstrips Is made. This somewhat 
Qllminates the problem of the child -s ability to readB 

The rGllabllity of the AVR has increased remarkably over the past 
year and arrangements are now undertmy for oxchange of the developnGntal 
units for production units, 

RESEARCH FIHDIMGS AM D QUESTIONS GENERATED 

1, Effects of pretest/posttest design. Because of the llTTiitatloas 

of the pre/post test design, it was desired to answer soveral 
questions concerning its use and effects as determined from the 
data. Below are the findings to date. 

A, Administration of pretest seems to depress posttest scores 



by about 101 when posttest scores of sinllar groups i, differing 
only In having had a pretest are compared. 

B, The same depressing effect Is noted whether data Is collected 
In home classrooms using a paper and pencil method or using 
the automated response system. 

c. Cotaments of children, frequently noted in testing situations, 
suggest that questions answered correctly on pretest are 
remeiiibered when prasented in posttest, 

D. niere is no significant difference in the abilities to 
identify numbers and letters and corract choices on 
posttest questions for young ffilR. 

E, Test items giving little, no, or negative gain with a small 
group (N s 5 to 25) tend to not change when a larger group 
(N » 125) is used. 



II, Automated vs No n-automated testing situations. Considerable 

concern has been eKpressed about the use of an automated system 
to collect data. In an effort to substantiata the student 



response system (SRS) the following findings occurred, 

A, Ho significant differences were found in the answer responses 
on multiple choice questions bett-yeen paper and pencil tests 
and the automated system. 
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B, Posttest scores tend to be about 10% higher using the SES 
than in a paper/pencil mode., however, pretest scores tend 
to be higher also. 

C. As noted in 1-B above, pretests have a dopressing effect on 
posttest scores; that is, when a pretest Is administered, 
the posttest scores tend to be lowor than when it is omitted, 

D, Children tend to be more eager to respond in the SRS than in 
their own classrooms. This could he caused by: 

1, movement from classroom to SRS, 

2, iiranediate confirmation of answers is given in SRS, 

3. equipTnent used in SRS requires manipulation, or 

4. other unidentified factors, 

E. Less distraction is recorded on attention profilos in SRS than 
in regular classrooms. Tliis results in less variahlillty of 
attention profiles, however, points of greatest non-attention 
can still he identified occurring at the same places in a 
media in either setting, 

P. Greater test management difficulties are noted in situations 
where chlldrQn are moved from regular classrooms to another 
site other than SRS room, 

^^fects of caD tlonlng and a u dio stlmull « These studies have 

just begun, nie findings thus far includes 

A. Sffl children make significantly more correct responses In an 
audio/viaual (both stimuli) setting than when either an 
auditory or visual only stimulus is administered, 
li, A significant difference in correct scores is noted when the 
auditory (sound track) of a film is heard prior to the sliowlng 
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of the film than in the other combination of visuals only 
or prior to shwjins with either or both auditory and visuals 
presented without any preliminary . 

C. Older S'lH (junior hiBh) tend to vocally verbalize the captions 
during a showing without being encouraged to do so. 

IV. Othor investlRatlons . During the course of the second year 

a number of short investigations have boon conducted with the 

foil owinc £ indings . 

A. childron tend to tire after about 25 framos of material 

presentod in an individual carrel. 

B. Selection of the correct answer in a multiplo-choice format 
requires a higher level skill than knowing the numbers 

and letters , 

C. Children can be trained to select correct responses in the 
SRS system. 

D. Tliore are common points of non-attention during the length of a 
given media when shown to a cross section of nvallablc EllH 
population. 

E. Inter-rater reliabilities for obsorvers attention arc 
quite high .95+ vyhen using video-taped subjects or the 
same classroom showing. 

F. Reliabilities of .60 - .80 are obtained from different 
snmplos of slinilar ago and academic groups using the scune 
or different observers of attention. 

G. Low reliabilities (of negative to +0.2) are obtained Mien 
the attention of young groups are compared to older groups . 
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OBJECTIVES FOR 1971 ~ 1972 

TliG project has established the following goals for 1971 - 1972. 

1. Tlie evaluation of 50 films and 100 filmstrips. 

2. Refinement of the model for cost ef f cctivene&s . 

3. Development of a classification systom, 

4. Dissemination of evaluativo information, 

5. Evaluation of some materials before purchase. 

6. Exploration of effective madia tGchnlqucs for 
educating the handicapped . 

7 . Utilization of computer sarviccs and data banks . 

8. Preparntion of personnel for field testing. 





- 22 - 



APPiroi: 




24 



EmUATIOfj SYSTEh 



ES " 1 




la 

1 recQ I 



t'-K 



Enter nedia 
into flies 

catalogue 




C , F, Eva 1 , 
Sys . Flow 




Trial 

A.O. 


Diagram 

#TR-A.O. 


f 

i 

1 

i \ 


\ y 



Process data 
thru computer 



CONPUTER OUT PUT 

\ attention 
profile 



\ 






file 



Classify 

content 

.to 

Att. 

Prof i le 



o 

ERIC 




( 

\ 

t 

CompU'^ j 

ter I 
Output I 



Attention 

graph 



■7 






Pre“Post I 
prlnt"Out / 



Process 
data thru 
Computer 




^ ' cf ! Eval . Sys 
I Flow Diagrarii 
^ #Tr - A/DI 



Prepa re 
Questions in 
SI I do form 



1 



Assign 
sscrh 



Schedule 
Trial A 



Seloot; 
Population 
or Trlil.. 



'? 



X, 



Instrument 
acceptable 
" X. ? 

yes 



make 

technical 

adjustment 



TRIAL B / 
testing / 
schedule / ^ 



Write Test 
Questions 




Conceptual I ze 
Object Ives 



attention 

graph 



pre " pos" 
test 

print oul 





- 23 - 





; END 


) 

/ 












j 









DIss" 
aminat&l 




\ 

« 

I 



I cf : Eva I . 
system flow 




ERIC 



EVALUATION SYSTEM TRIAL ATTENTION OBSERVATION TESTER'S ROLE #TR-AO 




Check 

test 

schedule 



pick up 
med I a and 
testing pkg. 



Bring media 
and test pkg 
to scheduled 
school 



Report to 
main office 



Pick up 

projection 

eauipment 



Bring media, 
testing pkg. anil 
proj . equip, to 
assigned ttest 
classroom 







show 




Record 


nied i a 




attending 


1 


behavior 


4 


^ 



Discuss the medle 
with the students 



X 





Interview individul 
students 
on attitudes 



HAiapH 



Record 
anecdotal 
comments 
(of tester) 
on presentat|* 
jonal 



Record 




comments 




of teacher 




concerning 




the media 






- 25 - 



EVALUATION SYSTEM-GROUP MODE-TRIAL ASB TESTER'S ROLE TR-B1 



Admin Is ter 
posftest 




File 

Ancedota I 
Comments 



o 

ERIC 



26 



EVALUATION SYSTEM TRIAL B 



INDIVinUAL MODE — TE''A£R‘S ROLE (TR-B2) 




o 

ERIC 



27 



29 




I 




EVALUATION SYSTEM! 
FLOW DIAGRAM - TR-Cl 




TABLE 1 



COMPUTEE USE FOR SECOND YEAB. 



MONTH 


TEMaWAL 


TIME 


PROCESS Il-T 


G UNITS 


COST 




(hours) 


(seconds) 








RADC* 


MARK II ** 


RADC 


imRIC 11 
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MMK II 
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42 


7 


906 


11 


1041 I 


92 


August 


18 


2 


460 


13 


637 


95 


September 


42 


2 


1350 


9 


1275 


104 


October 


55 


3 


1797 


11 


1454 


98 
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26 


3 
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56 


869 1 


1 125 


December 


39 


2 


3731 


17 


1361 i 


1 99 


January 


43 


1 


1876 


13 


1340 


90 


February 


45 


1 


1750 


5 


1325 


66 


March 


43 


1 


1680 


7 


1280 


49 


April 


38 


3 


1575 


23 


1190 


81 


May 


52 


3 


2120 


21 


1500 


85 


June 




4 


iflon 


53_„ 1 


0300 


lofi 


TOTALS 405 


27 


19,957 


239 


15,072 


j 1090 


GRAND TOTAL 


51 


.2 


20,196 


$16, 


162 



* - Rome Air Development Centered Computer 
** ~ General Elactrlc Time Sharing Mark II Service 
’ “ Value of service If obligated to pay 



TABLE II 



MAJOR PROCESSING PROGRiai USE 



MEDIA 

PROGR^iS , 


EViOLUATION SYSTBI 
TRIAL A TRIAL B 


STEP 

TRIAL C 


TOTAL 


70 Films 
Pre/Post 
ANALB 


169 

64 


262 

6 


11 

29 


442 

101 

543 


45 Filmstrips 










Pra/Post 


32 


8 




40 


MALB 


8 


- 


8 


16 










56 


52 Films 










Attention graphs 


82 






82 
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1. Films that ware presented by the project were generally useful teaching 
tools « 

2. Multiple-choice questioning is the most effective method of gaining information 
from my students. 

3. The project keeps lae informed about my sdude’^ts’ progress and its oxjn progress. 

4. Field testers have been courteous and friendly toward the students, 

5. The children looked forirard to the X7sekly film showings. 

6. My students were thoughtful and serious trtien rosponding to the pro and post tests. 

7. The questioning procedure used was not appropriate for my students, 
w. The films shotm by the project did not supplement my lesson plans, 

9, My students looked forward to x^eekly visits by the project field tester, 

' 10. Questions presented on the screen were easy to see. 

11. Language used in questioning has been too advanced for my studenta. 

12. The pretas t/posttest schemG was dull and boring for ny studeiits, 

13. Field testers were prompt and efficient. 

14. The films shotm generally did not take into account the ability level of ny 
students . 

15. Most of the filras shox-m by the project xmva of interest tc my studonts, 

16. My students did not participate in the project x#ith great interast. 

17. I enjoyed coopGrating X'Tith the project, 

18. I should be kept Informed about my students' progress and project progress, 

19. The weekly visits by project scaff member have been disruptive to my class. 

20. I would like zo participate in the project next year. 



20 Items scored 
34 Teachers poled 
2,59 Mean Response 
2,44 Standard deviation 
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Questions and 
Questioning technique 
Items 2 *6 ,7 ^ 10^11^ 12 
Frequency 
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Film Shown 
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Items IsSgSsWalS 
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Field Testers 
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Items 4 j9 5 13 9 19 
Frequency 


j 

3 


1 

12 


14 


65 


42 


Percentage 
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General Feelings 
toward Project 
Items 3 9 16 jl7 9 18 5 20 
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COMSULTMTS ' UTILIZATION 



NAME 



OmAMIZATlON 



ACTIVITY 



Mr 9 Richard Albano 



Dr« Burton Blatt 



Director g Project SAFE 
S.U.Ca Oneontaj IT.Y, 

Director 3 Division of 
Spoclal Education and 
Rehabilitation 3 
Syracuse University 



Discussions on flow-charting and 
PERTing of both evaluation systems 
and overall project management. 

Discussion on overall project 
goals and population character- 
istics , 



Dr, Donald Erickson 



Director, C,E<,C, 
Information Center on 
Exceptional Children 



Discussions on overall product 
with special emphasis on 
dissemination , 



Dr, Kenneth Flshell 



Associate Professor - 
Associate Director for 
Research and Develop- 
ment Center for Instruc- 
tional Communications 
Syracuse University 



On-going contributions to 
project managemGnt and the 
development of the evaluation 
system. 



Dr. Silas Halperin Associate Professor — Discussions on statistical 

Measurement, Evaluations analysis techniques, 
and Statistics 
School of Education 
Syracuse University 



Mr , Howard Lohr 



(Former EADC Association) Discussions on computer system 

problems and the Identification 
and Implementation of solutions . 



Dr, Mary Meeker A.ssociate Professor 

Guidance Center 
Loyola University 
Los Angeles, California 



Discussions on the Inclusion of 
the Structure of the Intellect 
model into the project's 
descriptors , 



Dr . WilllMi Meyer 



Dr. Gabriel Oflesh 



Mr, Casper Paulson 



^ Professor Psychology Discussions on the inclusion of 

School of Education attention gathering techniques 

Director of the S.U, in the evaluation system. 

Center of Early Childhood 
Education Center at S.U. 

Syracuse University 



Director, Center for Discussions on the overall 

Educational Technology evaluation Bystem 

Catholic University of 
America 



Teaching Research, 
Monmouth , Oregon 



Discussion on measuramant and 
evaluation problems. 
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ORGAWIEilTION 



ACTIVITY 



Jr, Hoing Pfoiffer 



Manager ^ Educational 
Technology Branch 
General Electric Research 
& Devolopment Conter 



Diacuaslon on overall project 
goals . 



Dr, Glenn Vergason 



Chnlrp'iana Department of 
Special Education 
Georgia State Unlveralty 
Atlanta j Georgia 



Discussion on overall project 
goals and characteristics of the 
student population. 



Dr, John Vinsonhaler 



Director j Infonnation 
Systems Laboratory 
Michigan State University 



Discussions on the Evaluation 
System. 



Dr, Timothy Weaver Research Fellow - Discussions on Project goals 

Educational Policy Research and activities. 

Center, Syracuse University 
lies oar ch Corporation 



Dr , Clarence Williams 



Professor - 

School of Education 

University of Rochester 



Discussion on project goals and 
the evaluation tnodel , 



General Electric Information Systems and Research and Development 
Corporation Personnel were utilized as consultants to specific operationol 
problema and by project representation at seminars. 
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